Entropy

#Entropy | 来源: 网络整理| 查看: 265

Next Article in Journal Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction Previous Article in Journal An Adaptive Hybrid Model for Wind Power Prediction Based on the IVMD-FE-Ad-Informer Journals Active Journals Find a Journal Proceedings Series Topics Information For Authors For Reviewers For Editors For Librarians For Publishers For Societies For Conference Organizers Open Access Policy Institutional Open Access Program Special Issues Guidelines Editorial Process Research and Publication Ethics Article Processing Charges Awards Testimonials Author Services Initiatives Sciforum MDPI Books Preprints.org Scilit SciProfiles Encyclopedia JAMS Proceedings Series About Overview Contact Careers News Blog Sign In / Sign Up Notice clear Notice

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

Continue Cancel clear

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess.

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Journals Active Journals Find a Journal Proceedings Series Topics Information For Authors For Reviewers For Editors For Librarians For Publishers For Societies For Conference Organizers Open Access Policy Institutional Open Access Program Special Issues Guidelines Editorial Process Research and Publication Ethics Article Processing Charges Awards Testimonials Author Services Initiatives Sciforum MDPI Books Preprints.org Scilit SciProfiles Encyclopedia JAMS Proceedings Series About Overview Contact Careers News Blog Sign In / Sign Up Submit Journals Entropy Volume 25 Issue 4 10.3390/e25040648

Submit to this Journal Review for this Journal Edit a Special Issue ► ▼ Article Menu Article Menu Academic Editor

Alessandro Giuliani Subscribe SciFeed Recommended Articles Related Info Link Google Scholar More by Author Links on DOAJ Jansma, A. on Google Scholar Jansma, A. on PubMed Jansma, A. /ajax/scifeed/subscribe Article Views Citations - Table of Contents Altmetric share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up ... Endorse textsms ... Comment Need Help? Support

Find support for a specific problem in the support section of our website.

Get Support Feedback

Please let us know what you think of our products and services.

Give Feedback Information

Visit our dedicated information section to learn more about MDPI.

Get Information clear JSmol Viewer clear first_page settings Order Article Reprints Font Type: Arial Georgia Verdana Font Size: Aa Aa Aa Line Spacing:    Column Width:    Background: Open AccessArticle Higher-Order Interactions and Their Duals Reveal Synergy and Logical Dependence beyond Shannon-Information by

Abel Jansma 1,2,3 ORCID

1 MRC Human Genetics Unit, Institute of Genetics & Cancer, University of Edinburgh, Edinburgh EH8 9YL, UK 2 Higgs Centre for Theoretical Physics, School of Physics & Astronomy, University of Edinburgh, Edinburgh EH8 9YL, UK 3 Biomedical AI Lab, School of Informatics, University of Edinburgh, Edinburgh EH8 9YL, UK Entropy 2023, 25(4), 648; https://doi.org/10.3390/e25040648 Received: 23 February 2023 / Revised: 6 April 2023 / Accepted: 7 April 2023 / Published: 12 April 2023 (This article belongs to the Section Information Theory, Probability and Statistics) Download Download PDF Download PDF with Cover Download XML Download Epub Browse Figures Review Reports Versions Notes

Abstract: Information-theoretic quantities reveal dependencies among variables in the structure of joint, marginal, and conditional entropies while leaving certain fundamentally different systems indistinguishable. Furthermore, there is no consensus on the correct higher-order generalisation of mutual information (MI). In this manuscript, we show that a recently proposed model-free definition of higher-order interactions among binary variables (MFIs), such as mutual information, is a Möbius inversion on a Boolean algebra, except of surprisal instead of entropy. This provides an information-theoretic interpretation to the MFIs, and by extension to Ising interactions. We study the objects dual to mutual information and the MFIs on the order-reversed lattices. We find that dual MI is related to the previously studied differential mutual information, while dual interactions are interactions with respect to a different background state. Unlike (dual) mutual information, interactions and their duals uniquely identify all six 2-input logic gates, the dy- and triadic distributions, and different causal dynamics that are identical in terms of their Shannon information content. Keywords: higher-order; information; entropy; synergy; triadic; Möbius inversions; Ising model; lattices 1. Introduction 1.1. Higher-Order InteractionsAll non-trivial structures in data or probability distributions correspond to dependencies among the different features, or variables. These dependencies can be present among pairs of variables, i.e., pairwise, or can be higher-order. A dependency, or interaction, is called higher-order if it is inherently a property of more than two variables and if it cannot be decomposed into pairwise quantities. The term has been used more generally to refer simply to complex interactions, as for example in [1] to refer to changes in gene co-expression over time; in this article, however, it is used only in the stricter sense defined in Section 2.The reason such higher-order structures are interesting is twofold. First, higher-order dependence corresponds to a fundamentally different kind of communication and interaction among the components of a system. If a system contains higher-order interactions, then its dependency structure cannot be represented by a graph and requires a hypergraph, where a single ‘hyperedge’ can connect more than two nodes. It is desirable to be able to detect and describe such systems accurately, which requires a good understanding of higher-order interactions. Second, higher-order interactions might play an important role in nature, and have been identified in various interaction networks, including genetic [2,3,4,5], neuronal [6,7,8,9,10], ecological [11,12,13], drug interaction [14], social [15,16,17], and physical [18,19] networks. Furthermore, there is evidence that higher-order interactions are responsible for the rich dynamics [20] or bistability [21] in biological networks; for example, synthetic lethality experiments have shown that the trigenic interactions in yeast form a larger network than the pairwise interactions [4].Despite this, purely pairwise descriptions of nature have been remarkably successful, which the authors of [22,23] attribute to the fact that there are regimes in terms of the strength and density of coupling among the variables within which pairwise descriptions are sufficient. Alternatively, it may be attributed to the fact that higher-order interactions have been understudied and their effects underestimated. Currently, perhaps the most promising method of quantifying higher-order interactions is information theory. The two most commonly used quantities are mutual information and its higher-order generalisation (used in, e.g., [24,25]) and the total correlation (introduced in [26] and recently used in [27]). However, one particular problem of interest that total correlation and mutual information do not address is that of synergy and redundancy. Given a set of variables with an nth-order dependency, what part of that is exclusively nth-order (called the synergistic part), and what part can be found in a subset of m x 0 , y 1 > y 0 , z 1 > z 0 , as all possible pairs necessarily sum to zero because I X Y Z ( x 0 → x 1 ; y 0 → y 1 ; z 0 → z 1 ) = − I X Y Z ( x 1 → x 0 ; y 0 → y 1 ; z 0 → z 1 ) . For the dyadic distribution, we have I X Y Z Dy ( 0 → 3 ; 0 → 3 ; 0 → 3 ) = log p ϵ 3 p ϵ 3 = 0 , while for the triadic distribution we have I X Y Z Tri ( 0 → 3 ; 0 → 3 ; 0 → 3 ) = log ϵ 4 p ϵ 3 = log ϵ p Thus, this particular 3-point interaction is zero for the dyadic distribution and negative for the triadic distribution. The sum over all three points (see Appendix A.4 for details) is provided by (76) I ¯ X Y Z Dy = log 1 = 0 (77) I ¯ X Y Z Tri = 64 log ϵ p That is, the additively symmetrised 3-point interaction is zero for the dyadic distribution and strongly negative for the triadic distribution. These two distributions, which are indistinguishable in terms of their information structure, are distinguishable by their model-free interactions, which accurately reflect the higher-order nature of the triadic distribution. 5. DiscussionIn this paper, we have related the model-free interactions introduced in [32] to information theory by defining them as Möbius inversions of surprisal on the same lattice that relates mutual information to entropy. We then invert the order of the lattice and compute the order-dual to the mutual information, which turns out to be a generalisation of differential mutual information. Similarly, the order-dual of interaction turns out to be interaction in a different context. Both the interactions and the dual interactions are able to distinguish all six logic gates by value and sign. Moreover, their absolute strength reflects the synergy within the logic gate. In simulations, the interactions were able to perfectly distinguish six kinds of causal dynamics that are partially indistinguishable to Pearson/partial correlations, causal graphs, and mutual information. Finally, we considered dyadic and triadic distributions constructed using pairwise and higher-order rules, respectively. While these two distributions are indistinguishable in terms of their Shannon information, they have different categorical MFIs that reflect the order of the construction rules.One might wonder why the interactions enjoy this advantage over entropy-based quantities. The most obvious difference is that the interactions are defined in a pointwise way, i.e., in terms of the surprisal of particular states, whereas entropy is the expected surprisal across an ensemble of states. Furthermore, the MFIs can be interpreted as interactions in an Ising model and as effective couplings in a restricted Boltzmann machine. As both these models are known to be universal approximators with respect to positive discrete probability distributions, the MFIs should be able to characterise all such distributions. What is not immediately obvious is that the kinds of interactions that characterise a distribution should reflect properties of that distribution, such as the difference between direct and indirect effects and the presence of higher-order structure. However, in the various examples covered in this manuscript the interactions turn out to intuitively align with properties of the process used to generate the data. While the stringent conditioning on variables not considered in the interaction might make it tempting to interpret an MFI as a causal or interventional quantity, it is important to be very careful when doing this. Assigning a causal interpretation to statistical inferences, whether in Pearl’s graphical do-calculus [49] or in Rubin’s potential outcomes framework [50], requires further (often untestable) assumptions and analysis of the system in order to determine whether a causal effect is identifiable and which variables to control for. In contrast, an MFI is simply defined by conditioning on all observed variables, makes no reference to interventions or counterfactuals, and does not specify a direction of the effect. While in a controlled and simple setting the MFIs can be expressed in terms of causal average treatment effects [32], a causal interpretation is not justifiable in general.Moreover, the stringency in the conditioning might worry the attentive reader. Estimating log p ( X = 1 , Y = 1 , T = 0 ) directly from data means counting states such as ( X , Y , T 1 , T 2 , … , T N ) = ( 1 , 1 , 0 , 0 , … 0 ) , which for sufficiently large N are rare in most datasets. Appendix A.1 shows how to use the causal graph to construct Markov blankets, making such estimation tractable when full conditioning is too stringent. In an upcoming paper, we address this issue by estimating the graph of conditional dependencies, allowing for successful calculation of MFIs up to the fifth order in gene expression data.One major limitation of MFIs is that they are only defined on binary or categorical variables, whereas many other association metrics are defined for ordinal and continuous variables as well. As states of continuous variables no longer form a lattice, it is hard to see how the definition of MFIs could be extended to include these cases.Finally, it is worth noting that the structure of different lattices has guided much of this research. That Boolean algebras are important in defining higher-order structure is not surprising, as they are the stage on which the inclusion–exclusion principle can be generalised [36]. However, it is not only their order-reversed duals that lead to meaningful definitions; completely unrelated lattices do as well. For example, the Möbius inversion on the lattice of ordinal variables from Figure 3 and the redundancy lattices in the partial information decomposition [28] both lead to new and sensible definitions of information-theoretic quantities. Furthermore, the notion of Möbius inversion has been generalised to a more general class of categories [51], of which posets are a special case. A systematic investigation of information-theoretic quantities in this richer context would be most interesting. FundingThis research was funded by Medical Research Council grant number MR/N013166/1.Data Availability StatementNo new data were created or analyzed in this study. Data sharing is not applicable to this article.AcknowledgmentsThe author is grateful for the insightful discussions of model-free interactions, causality, and RBMs with Ava Khamseh, Sjoerd Beentjes, Chris Ponting, and Luigi Del Debbio. The author also thanks John Baez for a helpful conversation on the role of surprisal in information theory. The author further thanks Sjoerd Beentjes for reading and commenting on an early version of this work, and the reviewers for their helpful suggestions. A.J. is supported by an MRC Precision Medicine Grant (MR/N013166/1).Conflicts of InterestThe funders had no role in the design of the study, in the collection, analysis, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.AbbreviationsThe following abbreviations are used in this manuscript: MIMutual InformationMFIModel-Free InteractionDAGDirected Acyclic GraphMBMarkov BlanketPIDPartial Information d=Decompositioni.i.d.independent and identically distributed Appendix A Appendix A.1. Markov BlanketsEstimating the interaction in Definition 2 from data involves estimating the probabilities of certain states occurring. While we do not have access to the true probabilities, we can rewrite the interactions in terms of expectation values. Note that all interactions involve factors of the type (A1) p ( X = 1 , Y = y ∣ Z = 0 ) p ( X = 0 , Y = y ∣ Z = 0 ) = p ( X = 1 ∣ Y = y , Z = 0 ) p ( X = 0 ∣ Y = y , Z = 0 ) (A2) = p ( X = 1 ∣ Y = y , Z = 0 ) 1 − p ( X = 1 ∣ Y = y , Z = 0 ) (A3) = E [ X ∣ Y = y , Z = 0 ] 1 − E [ X ∣ Y = y , Z = 0 ] because E [ X ∣ Z = z ] = ∑ x ∈ { 0 , 1 } p ( X = x ∣ Z = z ) x = p ( X = 1 ∣ Z = z ) This allows us to write the 2-point interaction, e.g., as follows: I i j = log E X i | X j = 1 , X ̲ = 0 E X i | X j = 0 , X ̲ = 0 1 − E X i | X j = 0 , X ̲ = 0 1 − E X i | X j = 1 , X ̲ = 0 Although expectation values are theoretical quantities, not empirical ones, sample means can be used as unbiased estimators to estimate each term in (A5). The stringent conditioning in this estimator can make the number of samples that satisfy the conditioning very small, which results in the estimates having large variance on different finite samples. Note that if we can find a subset of variables MB X i such that X i ⊥ ⊥ X k ∣ MB X i ∀ X k ∉ MB X i and i ≠ k (in causal language, a set of variables MB X i that d-separates X i from the rest), then we only have to condition on MB X i in (A5), reducing the variance of our estimator. Such a set MB X i is called a Markov Blanket of the node X i . There has recently been a certain degree of confusion around the notion of Markov blankets in biology, specifically with respect to their use in the free energy principle in neuroscience contexts. Here, a Markov blanket refers to the notion of a Pearl blanket in the language of [52]. Because conditioning on fewer variables should reduce the variance of the estimate by increasing the number of samples that can be used for the estimation, we are generally interested in finding the smallest Markov blanket. This minimal Markov blanket is called the Markov boundary.Finding such minimal Markov blankets is hard; in fact, because it requires testing each possible conditional dependency between the variables, we claim here (without proof) that it is causal discovery-hard, i.e., if such a graph exists it is at least as computationally complex as constructing a causal DAG consistent with the joint probability distribution. Appendix A.2. ProofsMarkov blankets are not only a computational trick; in theory, only variables that are in each other’s Markov blanket can share a nonzero interaction. To illustrate this, first note that the property of being in a variable’s Markov blanket is symmetric:Proposition A1 (symmetry of Markov blankets). Let X be a set of variables with joint distribution p ( X ) and let A ∈ X and B ∈ X such that A ≠ B . We denote the minimal Markov blanket of X by MB X . Then, A ∈ MB B ⇔ B ∈ MB A , and we can say that A and B are Markov-connected.Proof. Let Y = X \ { A , B } . Then, A ∉ MB B ⇒ p ( B ∣ A , Y ) = p ( B ∣ Y ) Consider that (A7) p ( A ∣ B , Y ) = p ( A , B ∣ Y ) p ( B ∣ Y ) (A8) = p ( B ∣ A , Y ) p ( A , ∣ Y ) p ( B ∣ Y ) (A9) = p ( A ∣ Y ) which means that B ∉ MB A . Because A ∉ MB B ⇔ B ∉ MB A holds, its negation holds as well, which completes the proof. □This definition of Markov connectedness allows us to state the following.Theorem A1 (only Markov-connected variables can interact). A model-free n-point interaction I 1 … n can only be nonzero when all variables S = { X 1 , … , X n } are mutually Markov-connected.Proof. Let X be a set of variables with joint distribution p ( X ) , let S = { X 1 , … , X n } , and let X ̲ = X \ S . Consider the definition of an n-point interaction among S: (A10) I 1 … n = ∏ i = 1 n ∂ ∂ X i log p ( X 1 , … , X n ∣ X ̲ = 0 ) (A11) = ∏ i = 1 n − 1 ∂ ∂ X i ∂ ∂ X n log p ( X 1 , … , X n ∣ X ̲ = 0 ) (A12) = ∏ i = 1 n − 1 ∂ ∂ X i log p ( X n = 1 ∣ X 1 , … , X n − 1 , X ̲ = 0 ) p ( X n = 0 ∣ X 1 , … , X n − 1 , X ̲ = 0 ) (A13) = ∏ i = 1 n − 1 ∂ ∂ X i log p ( X n = 1 ∣ S \ X n , X ̲ = 0 ) p ( X n = 0 ∣ S \ X n , X ̲ = 0 ) Now, if ∃ X j ∈ S such that X j ∉ MB X n , we do not need to condition on X j and can write this as (A14) I 1 … n = ∏ i = 1 n − 1 ∂ ∂ X i log p ( X n = 1 ∣ S \ { X j , X n } , X ̲ = 0 ) p ( X n = 0 ∣ S \ { X j , X n } , X ̲ = 0 ) (A15) = ∏ i = 1 i ≠ j n − 1 ∂ ∂ X i ∂ ∂ X j log p ( X n = 1 ∣ S \ { X j , X n } , X ̲ = 0 ) p ( X n = 0 ∣ S \ { X j , X n } , X ̲ = 0 ) (A16) = 0 as the probabilities no longer involve X j . Because X j was chosen arbitrarily, this must hold for all variables in S, which means that if any variable in S is not in the Markov blanket of X n then the interaction I S vanishes: S \ X n ⊄ MB X n ⇒ I S = 0 Furthermore, as the indexing we chose for our variables was arbitrary, this must hold for any re-indexing, which means that ∀ X i ∈ S : S \ X i ⊄ MB X i ⇒ I S = 0 This in turn means that all variables in S must be Markov-connected in order for the interaction I S to be nonzero. □Thus, knowledge of the causal graph aids estimation in two ways: it shrinks the variance of the estimates by relaxing the conditioning, and it identifies the interactions that could be nonzero.When knowledge of the causal graph is imperfect, it is possible to accidentally exclude a variable from a Markov blanket and thereby undercondition the relevant probabilities. The resulting error can be expressed in terms of the mutual information between the variables, as follows.Proposition A2 (underconditioning bias). Let S be a set of random variables with probability distribution p ( S ) , let X , Y , and let Z be three disjoint subsets of S. Then, omitting Y from the conditioning set results in a bias determined by (and linear in) the pointwise mutual information that Y = 0 provides about the states of X: I X ∣ Y Z − I X ∣ Z = ∏ i = 1 | X | ∂ ∂ x i pmi ( X = x , Y = 0 ∣ Z = 0 ) Proof. The pointwise mutual information (pmi) is defined as pmi ( X = x , Y = y ) = log p ( X = x , Y = y ) p ( X = x ) p ( Y = y ) Note that p ( X = x 1 ∣ Y = y , Z = z ) = p ( X = x 1 , Y = y ∣ Z = z ) p ( Y = y ∣ Z = z ) meaning that we can write p ( X = x 1 ∣ Y = y , Z = z ) = e pmi ( X = x 1 , Y = y ∣ Z = z ) p ( X = x 1 ∣ Z = z ) That is, not conditioning on Y = y results in an error in the estimate of p ( X = x 1 ∣ Y = y , Z = z ) that is exponential in the Z-conditional pmi of X and Y. However, consider the interaction among X, (A23) I X = I X ∣ Y Z = ∏ i = 1 | X | ∂ ∂ x i log p ( X = x ∣ Y = 0 , Z = 0 ) (A24) = ∏ i = 1 | X | ∂ ∂ x i log p ( X = x ∣ Z = 0 ) + pmi ( X = x , Y = 0 ∣ Z = 0 ) (A25) = I X ∣ Z + ∏ i = 1 | X | ∂ ∂ x i pmi ( X = x , Y = 0 ∣ Z = 0 ) That is, the error in the interaction as a result of not conditioning on the right variables is linear in terms of the difference between the pmi values of different states. □ Appendix A.3. Numerics of Causal StructuresTable A1, Table A2, Table A3, Table A4, Table A5 and Table A6 are taken from [48] with permission from the author, and list the precise values leading to Figure 4. From each graph, 100k samples were generated using p = 0.5 and σ = 0.4 . To quantify the significance value of the interactions, the data were bootstrap resampled 1k times, resulting in the definition of F as the fraction of resampled interactions having a different sign from the original interaction. The smaller F is, the more significant the interaction. Table

Table A1. Chain. Table A1. Chain. Entropy 25 00648 i003

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]4.2810.0000.7900.00.6350.000 × 10 + 0 0.5151[0, 2]0.0560.1170.6220.00.0312.261 × 10 − 23 0.3012[1, 2]4.2490.0000.7860.00.6280.000 × 10 + 0 0.5103[0, 1, 2]−0.0520.217NaNNaNNaNNaN0.300 Table

Table A2. Fork. Table A2. Fork. Entropy 25 00648 i004

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]4.2680.0000.7890.00.6340.000 × 10 + 0 0.5141[0, 2]4.2570.0000.7880.00.6320.000 × 10 + 0 0.5122[1, 2]−0.0140.3760.6220.00.0286.518 × 10 − 19 0.3003[0, 1, 2]0.0200.376NaNNaNNaNNaN0.300 Table

Table A3. Additive collider. Table A3. Additive collider. Entropy 25 00648 i005

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]2.1440.0000.3950.0000.5050.000 × 10 + 0 1.154 × 10 − 1 1[0, 2]−0.9890.000−0.0020.593−0.0705.172 × 10 − 109 2.059 × 10 − 6 2[1, 2]2.1440.0000.3950.0000.5050.000 × 10 + 0 1.154 × 10 − 1 3[0, 1, 2]0.0030.438NaNNaNNaNNaN−2.678 × 10 − 2 Table

Table A4. Multiplicative collider. Table A4. Multiplicative collider. Entropy 25 00648 i006

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]0.0320.1400.4270.0000.4780.000 × 10 + 0 1.403 × 10 − 1 1[0, 2]−2.1560.000−0.0050.145−40.0871.463 × 10 − 166 1.529 × 10 − 5 2[1, 2]0.0360.1090.4290.0000.4800.000 × 10 + 0 1.415 × 10 − 1 3[0, 1, 2]4.2370.000NaNNaNNaNNaN−1.150 × 10 − 1 Table

Table A5. Additive collider + chain. Table A5. Additive collider + chain. Entropy 25 00648 i007

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]2.1030.0000.7050.00.3620.00.3961[0, 2]3.2880.0000.7900.00.5990.00.5152[1, 2]2.1130.0000.7060.00.3640.00.3973[0, 1, 2]0.0500.162NaNNaNNaNNaN0.335 Table

Table A6. Multiplicative collider + chain. Table A6. Multiplicative collider + chain. Entropy 25 00648 i008

GenesInteractionFPearson cor.Pearson cor. pPartial cor.Partial cor. pMI0[0, 1]−0.0170.3420.7090.00.3650.00.4031[0, 2]2.0940.0000.7860.00.5960.00.5102[1, 2]−0.0570.0920.7070.00.3610.00.4013[0, 1, 2]4.3590.000NaNNaNNaNNaN0.293 Appendix A.4. Python Code for Calculating Categorical Dyadic and Triadic InteractionsdyadicStates = [[’a’, ’a’, ’a’], [’a’, ’c’, ’b’], [’b’, ’a’, ’c’], [’b’, ’c’, ’d’], [’c’, ’b’, ’a’], [’c’, ’d’, ’b’], [’d’, ’b’, ’c’], [’d’, ’d’, ’d’]] triadicStates = [[’a’, ’a’, ’a’], [’a’, ’c’, ’c’], [’b’, ’b’, ’b’], [’b’, ’d’, ’d’], [’c’, ’a’, ’c’], [’c’, ’c’, ’a’], [’d’, ’b’, ’d’], [’d’, ’d’, ’b’]] stateDict = {0: ’a’, 1: ’b’, 2:’c’, 3: ’d’} def catIntSymb(x0, x1, y0, y1, z0, z1, states): prob = lambda x, y, z: ’p’ if [x, y, z] in states else~’e’ num = prob(x1, y1, z1) + prob(x1, y0, z0) + prob(x0, y1, z0) + prob(x0, y0, z1) denom = prob(x1, y1, z0) + prob(x1, y0, z1) + prob(x0, y1, z1) + prob(x0, y0, z0) return (num, denom) numDy = ’’ denomDy = ’’ numTri = ’’ denomTri = ’’ for x0 in range(4): for x1 in range(x0+1, 4): for y0 in range(4): for y1 in range(y0+1, 4): for z0 in range(4): for z1 in range(z0+1, 4): nDy, dDy = catIntSymb(*[stateDict[x] for x in [x0, x1, y0, y1, z0, z1]], dyadicStates) numDy += nDy denomDy += dDy nTri, dTri = catIntSymb(*[stateDict[x] for x in [x0, x1, y0, y1, z0, z1]], triadicStates) numTri += nTri denomTri += dTri print(f’Dyadic interaction: log (p^{numDy.count("p") - denomDy.count("p")} e^{numDy.count("e") - denomDy.count("e")})’) print(f’Triadic interaction: log (p^{numTri.count("p") - denomTri.count("p")} e^{numTri.count("e") - denomTri.count("e")})’) // Output: >> Dyadic interaction: log (p^0 e^0) >> Triadic interaction: log (p^-64 e^64) ReferencesGhazanfar, S.; Lin, Y.; Su, X.; Lin, D.M.; Patrick, E.; Han, Z.G.; Marioni, J.C.; Yang, J.Y.H. Investigating higher-order interactions in single-cell data with scHOT. Nat. Methods 2020, 17, 799–806. [Google Scholar] [CrossRef] [PubMed]Lezon, T.R.; Banavar, J.R.; Cieplak, M.; Maritan, A.; Fedoroff, N.V. Using the principle of entropy maximization to infer genetic interaction networks from gene expression patterns. Proc. Natl. Acad. Sci. USA 2006, 103, 19033–19038. [Google Scholar] [CrossRef] [PubMed][Green Version]Watkinson, J.; Liang, K.C.; Wang, X.; Zheng, T.; Anastassiou, D. Inference of regulatory gene interactions from expression data using three-way mutual information. Ann. N. Y. Acad. Sci. 2009, 1158, 302–313. [Google Scholar] [CrossRef] [PubMed]Kuzmin, E.; VanderSluis, B.; Wang, W.; Tan, G.; Deshpande, R.; Chen, Y.; Usaj, M.; Balint, A.; Usaj, M.M.; Van Leeuwen, J.; et al. Systematic analysis of complex genetic interactions. Science 2018, 360, eaao1729. [Google Scholar] [CrossRef] [PubMed][Green Version]Weinreich, D.M.; Lan, Y.; Wylie, C.S.; Heckendorn, R.B. Should evolutionary geneticists worry about higher-order epistasis? Curr. Opin. Genet. Dev. 2013, 23, 700–707. [Google Scholar]Panas, D.; Maccione, A.; Berdondini, L.; Hennig, M.H. Homeostasis in large networks of neurons through the Ising model—Do higher order interactions matter? BMC Neurosci. 2013, 14, P166. [Google Scholar] [CrossRef][Green Version]Tkačik, G.; Marre, O.; Amodei, D.; Schneidman, E.; Bialek, W.; Berry, M.J. Searching for Collective Behavior in a Large Network of Sensory Neurons. PLoS Comput. Biol. 2014, 10, e1003408. [Google Scholar] [CrossRef][Green Version]Ganmor, E.; Segev, R.; Schneidman, E. Sparse low-order interaction network underlies a highly correlated and learnable neural population code. Proc. Natl. Acad. Sci. USA 2011, 108, 9679–9684. [Google Scholar] [CrossRef][Green Version]Yu, S.; Yang, H.; Nakahara, H.; Santos, G.S.; Nikolić, D.; Plenz, D. Higher-order interactions characterized in cortical activity. J. Neurosci. 2011, 31, 17514–17526. [Google Scholar]Gatica, M.; Cofré, R.; Mediano, P.A.; Rosas, F.E.; Orio, P.; Diez, I.; Swinnen, S.P.; Cortes, J.M. High-order interdependencies in the aging brain. Brain Connect. 2021, 11, 734–744. [Google Scholar]Sanchez, A. Defining Higher-Order Interactions in Synthetic Ecology: Lessons from Physics and Quantitative Genetics. Cell Syst. 2019, 9, 519–520. [Google Scholar] [CrossRef][Green Version]Grilli, J.; Barabás, G.; Michalska-Smith, M.J.; Allesina, S. Higher-order interactions stabilize dynamics in competitive network models. Nature 2017, 548, 210–213. [Google Scholar]Li, Y.; Mayfield, M.M.; Wang, B.; Xiao, J.; Kral, K.; Janik, D.; Holik, J.; Chu, C. Beyond direct neighbourhood effects: Higher-order interactions improve modelling and predicting tree survival and growth. Natl. Sci. Rev. 2021, 8, nwaa244. [Google Scholar]Tekin, E.; White, C.; Kang, T.M.; Singh, N.; Cruz-Loya, M.; Damoiseaux, R.; Savage, V.M.; Yeh, P.J. Prevalence and patterns of higher-order drug interactions in Escherichia coli. NPJ Syst. Biol. Appl. 2018, 4, 1–10. [Google Scholar]Alvarez-Rodriguez, U.; Battiston, F.; de Arruda, G.F.; Moreno, Y.; Perc, M.; Latora, V. Evolutionary dynamics of higher-order interactions in social networks. Nat. Hum. Behav. 2021, 5, 586–595. [Google Scholar]Cencetti, G.; Battiston, F.; Lepri, B.; Karsai, M. Temporal properties of higher-order interactions in social networks. Sci. Rep. 2021, 11, 1–10. [Google Scholar]Grabisch, M.; Roubens, M. An axiomatic approach to the concept of interaction among players in cooperative games. Int. J. Game Theory 1999, 28, 547–565. [Google Scholar]Matsuda, H. Physical nature of higher-order mutual information: Intrinsic correlations and frustration. Phys. Rev. E 2000, 62, 3096. [Google Scholar]Cerf, N.J.; Adami, C. Entropic bell inequalities. Phys. Rev. A 1997, 55, 3371. [Google Scholar]Battiston, F.; Amico, E.; Barrat, A.; Bianconi, G.; Ferraz de Arruda, G.; Franceschiello, B.; Iacopini, I.; Kéfi, S.; Latora, V.; Moreno, Y.; et al. The physics of higher-order interactions in complex systems. Nat. Phys. 2021, 17, 1093–1098. [Google Scholar]Skardal, P.S.; Arenas, A. Higher order interactions in complex networks of phase oscillators promote abrupt synchronization switching. Commun. Phys. 2020, 3, 1–6. [Google Scholar]Merchan, L.; Nemenman, I. On the Sufficiency of Pairwise Interactions in Maximum Entropy Models of Networks. J. Stat. Phys. 2016, 162, 1294–1308. [Google Scholar] [CrossRef][Green Version]Tkacik, G.; Schneidman, E.; Berry, M.J., II; Bialek, W. Ising models for networks of real neurons. arXiv 2006, arXiv:q-bio/0611072. [Google Scholar]Margolin, A.A.; Nemenman, I.; Basso, K.; Wiggins, C.; Stolovitzky, G.; Favera, R.D.; Califano, A. ARACNE: An algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context. BMC Bioinform. 2006, 7, S7. [Google Scholar] [CrossRef][Green Version]Nemenman, I. Information theory, multivariate dependence, and genetic network inference. arXiv 2004, arXiv:q-bio/0406015. [Google Scholar]Watanabe, S. Information theoretical analysis of multivariate correlation. IBM J. Res. Dev. 1960, 4, 66–82. [Google Scholar]Rosas, F.E.; Mediano, P.A.; Luppi, A.I.; Varley, T.F.; Lizier, J.T.; Stramaglia, S.; Jensen, H.J.; Marinazzo, D. Disentangling high-order mechanisms and high-order behaviours in complex systems. Nat. Phys. 2022, 18, 476–477. [Google Scholar]Williams, P.L.; Beer, R.D. Nonnegative decomposition of multivariate information. arXiv 2010, arXiv:1004.2515. [Google Scholar]Wibral, M.; Priesemann, V.; Kay, J.W.; Lizier, J.T.; Phillips, W.A. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain Cogn. 2017, 112, 25–38. [Google Scholar]Jaynes, E.T. Information theory and statistical mechanics. Phys. Rev. 1957, 106, 620. [Google Scholar] [CrossRef]Nguyen, H.C.; Zecchina, R.; Berg, J. Inverse statistical problems: From the inverse Ising problem to data science. Adv. Phys. 2017, 66, 197–261. [Google Scholar] [CrossRef][Green Version]Beentjes, S.V.; Khamseh, A. Higher-order interactions in statistical physics and machine learning: A model-independent solution to the inverse problem at equilibrium. Phys. Rev. E 2020, 102, 053314. [Google Scholar] [CrossRef]Glonek, G.F.; McCullagh, P. Multivariate logistic models. J. R. Stat. Soc. Ser. B (Methodol.) 1995, 57, 533–546. [Google Scholar]Bartolucci, F.; Colombi, R.; Forcina, A. An extended class of marginal link functions for modelling contingency tables by equality and inequality constraints. Stat. Sin. 2007, 17, 691–711. [Google Scholar]Bateson, G. Steps to an Ecology of Mind; Chandler Publishing Company: San Francisco, CA, USA, 1972. [Google Scholar]Stanley, R.P. Enumerative Combinatorics, 2nd ed.; Cambridge Studies in Advanced Mathematics; Cambridge University Press: Cambridge, UK, 2011; Volume 1. [Google Scholar]Rota, G.C. On the foundations of combinatorial theory I. Theory of Möbius functions. Z. Wahrscheinlichkeitstheorie Verwandte Geb. 1964, 2, 340–368. [Google Scholar]Bell, A.J. The co-information lattice. In Proceedings of the Fifth International Workshop on Independent Component Analysis and Blind Signal Separation, ICA, Nara, Japan, 1–4 April 2003; Volume 2003. [Google Scholar]Galas, D.J.; Sakhanenko, N.A. Symmetries among multivariate information measures explored using Möbius operators. Entropy 2019, 21, 88. [Google Scholar]Galas, D.J.; Sakhanenko, N.A.; Skupin, A.; Ignac, T. Describing the complexity of systems: Multivariable “set complexity” and the information basis of systems biology. J. Comput. Biol. 2014, 21, 118–140. [Google Scholar]Galas, D.J.; Kunert-Graf, J.; Uechi, L.; Sakhanenko, N.A. Towards an information theory of quantitative genetics. J. Comput. Biol. 2019, 28, 527–559. [Google Scholar] [CrossRef]Freund, Y.; Haussler, D. Unsupervised learning of distributions on binary vectors using two layer networks. Adv. Neural Inf. Process. Syst. 1991, 4, 912–919. [Google Scholar]Le Roux, N.; Bengio, Y. Representational power of restricted Boltzmann machines and deep belief networks. Neural Comput. 2008, 20, 1631–1649. [Google Scholar] [PubMed]Montufar, G.; Ay, N. Refinements of universal approximation results for deep belief networks and restricted Boltzmann machines. Neural Comput. 2011, 23, 1306–1319. [Google Scholar] [PubMed][Green Version]Cossu, G.; Del Debbio, L.; Giani, T.; Khamseh, A.; Wilson, M. Machine learning determination of dynamical parameters: The Ising model case. Phys. Rev. B 2019, 100, 064304. [Google Scholar]Krumsiek, J.; Suhre, K.; Illig, T.; Adamski, J.; Theis, F.J. Gaussian graphical modeling reconstructs pathway reactions from high-throughput metabolomics data. BMC Syst. Biol. 2011, 5, 21. [Google Scholar]James, R.G.; Crutchfield, J.P. Multivariate dependence beyond Shannon information. Entropy 2017, 19, 531. [Google Scholar]Jansma, A. Higher-Order Interactions in Single-Cell Gene Expression. Ph.D. Thesis, University of Edinburgh, Edinburgh, UK, 2023. [Google Scholar]Pearl, J. Models, Reasoning and Inference; Cambridge University Press: Cambridge, UK, 2000; Volume 19. [Google Scholar]Imbens, G.W.; Rubin, D.B. Causal Inference in Statistics, Social, and Biomedical Sciences; Cambridge University Press: Cambridge, UK, 2015. [Google Scholar]Leinster, T. Notions of Möbius inversion. Bull. Belg. Math. Soc.-Simon Stevin 2012, 19, 909–933. [Google Scholar]Bruineberg, J.; Dołęga, K.; Dewhurst, J.; Baltieri, M. The emperor’s new Markov blankets. Behav. Brain Sci. 2022, 45, e183. [Google Scholar] Entropy 25 00648 g001 550

Figure 1. The lattices associated with P ( { X , Y } ) (left) and P ( { X , Y , Z } ) (right) ordered by inclusion. An arrow b → a indicates a

【本文地址】

公司简介

联系我们